Diametrical Risk Minimization: theory and computations

The theoretical and empirical performance of Empirical Risk Minimization (ERM) often suffers when loss functions are poorly behaved with large Lipschitz moduli spurious sharp minimizers. We propose analyze a counterpart to ERM called Diametrical (DRM), which accounts for worst-case risks within neighborhoods in parameter space. DRM has generalization bounds that independent convex as well nonconvex problems it can be implemented using practical algorithm based on stochastic gradient descent. Numerical results illustrate the ability find quality solutions low error risk landscapes from benchmark neural network classification corrupted labels.